Currently, most digital cameras use a zoom taking lens and a single color image sensor to capture still and motion images. The captured images are then processed to produce digital image files, which are stored in a digital memory in the camera. The digital image files can then be transferred to a computer, displayed, printed, and shared via the Internet.
In order to capture sharp images of moving subjects, a digital camera needs to provide a precise automatic lens focusing system (i.e., an autofocus system). The autofocus system must be capable of quickly obtaining the correct focus in order to minimize the “shutter delay” between the time the shutter button is pressed and the still image is captured. The autofocus system must also work in a continuous image capture mode wherein video images are captured. For instance, in a video mode the focus should be adjusted in real-time while video images are being continuously captured.
Many digital cameras and scanners capture images using an image sensor and a taking lens system with an adjustable focus. Typically, the focus distance of such an adjustable focus taking lens system can automatically be set to one of a plurality of different settings by sensing, control, and drive systems, which are adapted to provide optimal focus of what is determined to be a subject area in a scene. Lens systems that provide automatically adjustable focus settings based on a focus measurement and an adjustable focus lens are referred to herein as autofocus systems. Digital cameras typically use one of two types of autofocus systems: a rangefinder system and a “through-the-lens” focus system.
A rangefinder system uses rangefinding sensors such as a sonic rangefinder or a dual lens rangefinder to determine the distance from a camera to one or more portions of a scene within a field of view of the rangefinder system. A sonic rangefinder measures the phase offset between a projected sonic signal and a reflected sonic signal to infer the distance to objects in the scene. Dual lens rangefinders contain two lenses that are separated by a distance along with two matching sensor areas that capture matched pairs of images. Dual lens rangefinders are commonly used on digital cameras in the form of dual lens rangefinder modules which contain two lenses separated by a distance along with two matching sensor areas that capture matched pairs of low resolution images.
Common dual lens rangefinder-based autofocus systems include active and passive systems. Active systems actively project light onto the scene, while passive systems work with the available light from the scene. Dual lens rangefinder modules can be purchased from Fuji Electric in several models such as the FM6260W. A dual lens rangefinder module for optical apparatus such as a camera is described in U.S. Pat. No. 4,606,630, which was issued to Haruki et al. on Aug. 19, 1986 (and assigned to Fuji Electric). According to the description of the prior art in this patent, matched pairs of low resolution images are analyzed for correlation between the two images to determine the offset between the two images caused by the separation between the two lenses.
A diagram illustrative of the principle of the operation of a conventional rangefinder is shown herein in FIG. 27. In that diagram, light from an object 151 is incident on two small lenses 152 and 153 which have a sufficiently short focal length f that light rays received from the object through different spaced paths 154 and 155 produce corresponding spaced images 157 and 158 in a focal plane 156 which is common to the lenses 152 and 153. When the object 151 is at an infinite distance, the centers of the images 157 and 158 are located at reference positions 170 and 180 in FIG. 27, but when the object 151 is located at a closer distance, the centers of the images are shifted apart to positions 171 and 181. If the distance by which the images 157 and 158 are shifted from the reference positions 170 and 180 are designated x1 and x2, respectively, then the total shift x may be expressed as follows:x=x1+x2=b·f/d Thus, the distance d to the object 151 can be measured by d=b·f/x. In this case, b is the distance between the optical axes of the small lenses, that is, the base length. To obtain the shifted amounts x1 and x2, or the sum x of both, two optical sensor arrays 190 and 191 are provided in the focal plane 156 as shown in FIG. 27. These optical sensor arrays each comprise a plurality of optical sensors, for instance CCD devices, and an analog photoelectric signal is generated by each optical sensor corresponding to the light intensity at the portion of the image which is incident on the sensor. Haruki et al. shows a conventional circuit, as well as a higher speed rangefinding circuit according to the patent, for obtaining the sum x of the shifted distances by comparing two image signal trains comprising the digital image signals from the left and right optical sensor arrays.
Basically, the offset information x is used along with the lens separation distance b and the focal length f to calculate the distanced to the scene by triangulation. The calculated distance d to the scene is used to guide the positioning of an adjustable focus lens to produce the best image quality. As known in the prior art, this adjustment may be based on a calibration curve established between the distance to the scene as measured by the dual lens rangefinder module and a series of best focused images as produced by a “through the lens” autofocus system. The calibration curve is stored as an equation or a look-up table in a microprocessor in the camera.
Rangefinder-based autofocus systems have the advantage of being very fast, some having a response time that can be in the range of 0.01-0.05 second. However, the focus quality produced by some rangefinder-based autofocus systems can vary when they are used in different operating conditions. For example, sonic autofocus systems cannot focus through a glass window as the glass stops the projected sonic signal, thereby causing the autofocus system to focus onto the glass. In the case of a dual lens rangefinder autofocus system, the accuracy of dual lens rangefinders are typically influenced by changes in environmental conditions such as temperature and/or humidity. The problem with dual lens rangefinder modules is that the calibration between the dual lens rangefinder module and the adjustable focus lens position is not stable within the normal operating environment for digital cameras. Environmental conditions such as changes in temperature and humidity can cause the distance to the portion of the scene as measured by the dual lens rangefinder module to change by over 10%. In addition, the measured position of the adjustable focus taking lens in the adjustable focus taking lens system is prone to environmentally induced changes as well so that inaccuracies are produced in the control system for the adjustable focus lens. Consequently, dual lens rangefinder modules are not typically used independently for autofocus in digital cameras but instead are used as a rough focus adjustment that is supplemented by a “through the lens” autofocus system.
Alternatively, the “through-the-lens” autofocus system determines a focus state through an analysis of a series of autofocus images captured with the adjustable focus lens system positioned at a plurality of different focus distances. For example, in a typical “through-the-lens” autofocus system a plurality of autofocus images (e.g., 5-20) are captured with the adjustable focus lens in a series of different positions in a so-called “hill climb” method. This type of autofocus is known as “hill climbing” autofocus because it generates a sequence of values that increase in level until they pass over a peak, i.e., a “hill”. In other words, the lens focus position is adjusted automatically until the contrast of the edge detail in the image, or a particular area of the image, is maximized. For instance, the contrast present in each of the autofocus images is compared and the autofocus image with the greatest contrast is deemed to have been captured with the best focus conditions (often the best focus lens position is further refined by interpolating the contrast values between images).
In order to decrease focusing response time without sacrificing focusing precision, it is common to use filters to separate not only the higher frequency component of the video signal, but also the lower frequency component. For example, a lens may be quickly driven in coarse adjustment steps in a low frequency range furthest from the maximum focus, and then driven in finer adjustment steps in a high frequency range nearer to the maximum focus. A flow diagram of a conventional “hill climbing” contrast autofocus algorithm is shown in FIG. 28. This algorithm uses the “hill climbing” contrast autofocus method discussed above and shown in the diagram of FIG. 29, which illustrates the relationship between the focus value obtained from the filters and the lens position. In FIG. 29, the abscissa indicates the focusing position of a lens along a distance axis, the ordinate indicates the focusing evaluation value, and the curves A and B indicate the focusing evaluation values for high and low frequency components, respectively, relative to a particular in-focus position P.
Referring to the flow diagram of FIG. 28, the best starting point for the algorithm depends on the hyperfocal distance of the current lens setting, which is a function of the focal length setting and the f-number. A distance of about 2 meters is typically a good starting point. Then a low frequency bandpass filter is loaded (stage 197) and the focus values are read out. The algorithm employs a comparison stage 198 to set the direction of lens adjustment toward increasing focus values, and to determine when the lens is stepped over the “hill”. The depth of field, which depends on the present focal length and f-number, sets the number of steps, i.e., the next near focus position, which should be taken before capturing the next frame when using the low frequency bandpass filter. Once the peak of the hill is passed (curve B in FIG. 29), a high frequency bandpass filter is loaded (stage 199), and the lens is moved in the opposite direction until the peak of the higher “hill” is found (curve A in FIG. 29). The peak focus value may use either the weighted average or peak value from numerous pixels.
“Through-the-lens” autofocus systems are very accurate since they measure focus quality directly from autofocus images captured with the high quality taking lens. Unfortunately, “through-the-lens” autofocus systems can be relatively slow in determining a focus setting due to the large number of autofocus images that must be captured and compared. For example, “through-the-lens” autofocus systems can take as long as 0.5-2.0 seconds to determine focus conditions.
Accordingly, in some digital cameras, the two types of autofocus systems are used together in a hybrid system in which the rangefinder based autofocus system is used to provide a fast estimation of the adjustable focus lens location that is then followed by the use of the “through-the-lens” autofocus system to refine the focus setting. For example, U.S. Pat. No. 6,864,474, entitled “Focusing Apparatus for Adjusting Focus of an Optical Instrument” and which issued Mar. 8, 2005 in the name of Misawa, describes the coordinated use of a rangefinder-based autofocus system with a “through-the-lens” autofocus system. In Misawa, the focus position of the adjustable focus taking lens is determined by both the rangefinder-based autofocus system and the “through-the-lens” autofocus system. The difference between the adjustable focus taking lens position determined by the rangefinder-based autofocus system and the adjustable focus taking lens position determined by the “through-the-lens” autofocus system is stored for future reference. In subsequent image capture episodes, the stored difference information is used to refine the number of autofocus images captured and analyzed by the “through-the-lens” autofocus system in the “hill climb” method to determine the adjustable focus lens position for best focus, thereby reducing the number of autofocus images captured and processed in cases where the rangefinder system is accurate and increasing the number of autofocus images captured and processed in cases where the rangefinder is inaccurate. However, the method described by Misawa assumes that the performance of the rangefinder, adjustable focus taking lens system and control system are consistent over time, do not fluctuate with variations in environmental conditions, and do not otherwise change or drift over time.
Once an image is in focus, the “hill climb” method typically operates over incremental distances near the subject presently focused upon. Then, in refocusing an image, the “hill climb” method typically determines whether any lens movement is stepping “up or down the hill” and resets the lens for a new maximum. In practice, this means that, if the lens movement is stepping “down the hill”, the lens motion is immediately reversed so as to seek the new maximum for the existing subject. This is a particular problem in video focusing, where a new subject at some distance away from the present subject may come into the image and never be detected by the “hill climb” method—even where the new subject may present a greater “hill” in terms of contrast values. One way of responding to this problem is referred to as “whole way” autofocusing, where the auto focus module looks over all the distances discernible by the taking lens before deciding upon a focus position.
Commonly assigned U.S. Pat. No. 6,441,855 describes a “whole-way” autofocusing method, where a focusing device includes a movable focusing lens adapted to be moved to different positions across the entire focusing range, a conversion element for converting light incident on and transmitted through the focusing lens into a signal, and a lens driving mechanism for moving the focusing lens. The focusing device further includes a focus evaluation value calculation unit for calculating a focus evaluation value for each position of the focusing lens based on the signal from the conversion element. The focus evaluation value calculation unit extracts only the signals corresponding to the pixels in a focus area defined, e.g., at the center of an image, which is further divided into nine “tiles”, that is, blocks that are obtained by dividing the focus area into a small number of rows and columns used as observation areas.
In calculating the focus evaluation values, a determination is first made as to whether or not the calculation of the focus evaluation values has been repeated a certain number of times, e.g., ten times, for different distance settings. When the determination is negative, the focusing lens is moved by a preset step width, and the calculation is repeated. Thus, the focusing lens is always moved stepwise from an infinite far position to a nearest position, and for each step of movement, a focus evaluation value is calculated for each tile. These calculations are performed for the respective tiles, to thereby obtain the focus evaluation values for ten lens positions for each of the nine tiles, including all of the peaks that are found across the total distance. Using the ten total sums obtained for the respective lens positions as the focus evaluation values, the focusing lens position producing the maximum peak is determined as the in-focus lens position. A lens driving output is then applied to the lens driving mechanism so that the lens moves to the determined in-focus position.
In order to provide a small size digital camera with a large “optical zoom range”, a digital camera can use multiple image sensors with different focal length lenses, as described in commonly assigned U.S. patent application Ser. No. 11/062,174, entitled “Digital Camera Using Multiple Lenses and Image Sensors to Provide an Improved Zoom Range”, which was filed Feb. 18, 2005 in the names of Labaziewicz et al., the disclosure of which is incorporated herein by reference. For example, the Kodak Easyshare V610 dual lens digital camera includes a 38-114 mm (35 mm equiv.) f/3.9-f/4.4 lens and a 130-380 mm (35 mm equiv.) f/4.8 lens, in order to provide a 10× optical zoom range. However, in both this patent application and product, only one of the two image sensors is used at a time. The two image sensors do not simultaneously capture images.
U.S. Patent Application Publication No. US 2003/0020814, which was published Jan. 30, 2003, discloses an image capturing apparatus having a plurality of capturing optical systems, each coupled to a CCD image sensor, including a first system having a shorter focal length and a second system having a longer focal length. In the various embodiments described in this disclosure, the two lenses can provide different focal lengths ranges, including one system with a fixed-focus lens and the other system with a zoom lens, or they can both be fixed focus lenses set to two different focus distance settings. In each case, rather than obtaining user input, a selection unit automatically selects the capture signals from one of the capturing optical systems based on capture conditions, such as measured distance or luminance, determined by a capture condition acquiring unit. The autofocus for these systems is provided using a separate distance sensor. Neither of the two CCD image sensors are used for the autofocusing operation.
U.S. Patent Application Publication No. US 2003/0160886, which was published Aug. 23, 2003, discloses a digital camera having two photographing systems that are independent of each other. One embodiment shows one system including a monofocal “ordinary mode” lens and the other system including a zoom “telescopic mode” lens, each generating an image. An operator-actuated change over switch determines which image is to be recorded. Autofocus is also disclosed in connection with the separate photographing systems, where a “hill-climb” contrast comparison technique used in one system complements a “hill-climb” contrast comparison technique used in the other system. When it is desired to capture an image from the telescopic mode optical system, a rough autofocus search (where a stepping motor may be driven at intervals of several steps) is made by the ordinary mode optical system (where the focal depth is relatively large). This rough search results in a reduced focal distance range that includes the focusing position. Using the focal distance range information provided by the ordinary mode optical system, the telescopic mode optical system is driven to an autofocus search start position at one end of the reduced focal distance range. Then, a fine autofocus search is performed by the telescopic mode optical system (where the focal depth is relatively shorter), but only in the reduced focal distance range determined by the ordinary mode autofocus search. (When it is desired to capture an image from the ordinary mode optical system, the autofocus search is made solely by the ordinary mode optical system, with the telescopic mode optical system playing no part in the autofocus search.)
In another embodiment in U.S. Patent Application Publication No. US 2003/0160886, which does not depend on the rough vs. fine search mentioned above, a “hill climb” contrast comparison search is performed while the focusing lens of a first optical system is driven stepwise so as to move from an infinite distance setting toward a closest distance position, and a second “hill climb” contrast comparison search is performed while the focusing lens of a second optical system is driven stepwise from the closest position toward the infinite setting. This procedure continues until a maximum contrast position is located, although neither system ordinarily needs to move through its entire range. This tends to reduce the time period for detecting a focusing position. In this embodiment, each of the optical systems could be used for capturing an image and for focus adjustment, or one optical system could be employed for capturing an image and focus adjustment and the other optical system could be devoted only to focus adjustment of the image-capturing optical system. In another embodiment, in the case where the non-capturing optical system determines the focusing position first, the capturing optical system is driven to that position and a fine adjustment is then made by the capturing optical system.
A problem with these prior art systems is that either a separate autofocus sensor must be used (thus increasing the cost) or else there is typically a significant “shutter delay” as the autofocus is performed using the same sensor that is used to capture the image. Moreover, the separate autofocus sensor is usually a rangefinder and, as mentioned above, the calibration between the dual lens rangefinder module and the adjustable focus lens position is not stable within the normal operating environment for digital cameras. Where the autofocus is performed with the “through-the-lens” taking system, the process can be relatively slow in determining a focus setting due to the large number of autofocus images that must be captured and compared. The problem can be somewhat alleviated according to the aforementioned U.S. Patent Application Publication No. US 2003/0160886, but difficulties remain in rapidly achieving focus as the subject changes or moves, or in rapidly interchanging the focusing requirements of the optical systems when the operator elects to change the capture function from one photographing system to the other.
A special problem arises during video capture, where the autofocus images are derived from the same series of still images or frames that compose the video images. Consequently, the process of autofocusing may cause 5-20 or more out of focus video images to be produced in the video each time the scene changes. As a result, during video capture with pan movements of the camera where the scene changes continuously, large portions of the video are actually out of focus as the autofocus system hunts for proper focus. A further problem is that during video capture, many of the frames are out of focus due to the use of an autofocus system that uses the “hill climb method” to focus.
In copending, commonly-assigned U.S. patent application Ser. No. (docket 93231), entitled “Camera Using Multiple Lenses and Image Sensors to Provide Improved Focusing Capability” and filed on even date herewith in the names of John Border et al., the aforementioned problems are addressed in a multi-lens digital camera in which two (or more) image capture stages are used to separately capture images of the same scene so that one image capture stage can be used for autofocus while the other(s) is used for capturing a still image or a video. This provides precise, rapid autofocus in both still and video modes without unduly increasing the size or cost of the digital camera.
In one of the embodiments disclosed in that application, such a camera provides two separate image capture stages that are operable in a rangefinder configuration to autofocus the stage used for capturing the image. Besides autofocus, this approach offers a number of other possibilities for using the images to identify and map distances to respective portions of the image field. What is therefore needed is a digital camera with dual capture systems that provides an improved and extended rangefinding capability by utilizing both images to support a range mapping function.